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Presentation  Objectives 


GAO 

Accountability  *  Integrity  *  Reliability 


•  Provide  observations  on  the  Department’s  efforts  to  manage  its 
2009  portfolio  of  major  weapon  system  programs,  performance  of 
newer  programs,  and  ability  to  deliver  to  the  warfighter  on  time 

•  Analyze  outcomes  and  knowledge  attained  at  key  junctures  in  the 
acquisition  process  for  a  subset  of  the  42  programs  primarily  still  in 
development 

•  Gather  data  on  other  factors  that  might  impact  program  stability 
and  outcomes  such  as:  cost  estimating,  requirement  setting, 
software  management,  and  program  office  staffing 

•  Provide  an  update  on  any  impacts  from  DOD  acquisition  policy 
changes  and  Congressional  acquisition  reform  legislation 
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PORTFOLIO  LEVEL 


Accountability  *  Integrity  *  Reliability 


Observations  on  DOD’s  Fiscal  Year  2009 
Major  Defense  Acquisition  Program  Portfolio 


•  DOD’s  major  defense  acquisition  portfolio  grew  to  102  programs 
in  July  2009  -  a  net  increase  of  6  programs  since  December  2007. 

•  Eighteen  programs  in  the  portfolio  are  newly  designated  major 
defense  acquisition  programs.  The  total  acquisition  cost  of  the 
thirteen  new  programs  with  cost  data  is  over  $72  billion. 

•  Twelve  programs  with  a  cost  of  $48  billion,  including  $7  billion  in 
cost  growth  since  their  first  estimate,  left  the  portfolio.  If  FCS  is 
included,  these  numbers  increase  to  $179  billion  and  $48  billion 
respectively. 

•  The  lack  of  complete  Selected  Acquisition  Reports  in  2009 
precluded  a  definitive  analysis  of  the  overall  cost  and  schedule 
performance  of  the  portfolio. 


PORTFOLIO  LEVEL 


Accountability  *  Integrity  *  Reliability 


Observations  on  DOD’s  Fiscal  Year  2009 
Major  Defense  Acquisition  Program  Portfolio 


The  Secretary  of  Defense’s  fiscal  year  2010  budget  recommended 
canceling  or  curtailing  all  or  part  of  at  least  a  half  dozen  major  defense 
acquisition  programs,  including  CSAR-X,  DDG  1000,  FCS,  and  VH-71. 


Weapon  system  Secretary’s  comments 


Recommended 

terminations 

VH-71  Presidential  Helicopter 

Plan  to  develop  options  for  new  program 

Combat  Search  and  Rescue  Helicopter 

Plan  to  reexamine  requirements 

Next-Generation  Bomber 

Will  not  initiate  new  development  program  without 
better  understanding  of  requirements  and  technology 

Future  Combat  Systems-Manned 
Ground  Vehicles 

Plan  to  reevaluate  requirements,  technology,  and 
approach  before  relaunching  and  recompeting  program 

Transformational  Satellite 

Plan  to  buy  two  more  AEHF  satellites  as  alternative 

Ballistic  Missile  Defense-Multiple  Kill 
Vehicle 

Plan  to  reexamine  requirements;  no  mention  of  new 
program 

Recommended 
end  of  production 

C-1 7 

Recommended  ending  production  at  205  aircraft 

DDG  1000 

Recommended  ending  production  at  3  ships 

F-22 

Recommended  ending  production  at  187  aircraft. 

Source:  GAO  analysis  of  DOD  data. 
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PROGRAM  LEVEL 


Accountability  *  Integrity  *  Reliability 


A  Knowledge-Based  Approach  is  Key  to 
Successful  Program  Outcomes 


Material  Development 
Decision 


Knowledge 

Based 

Model 


5  to  6  years  or  less 


Development  Start 
B 


Production  Start 


Technology 

development 

Product  development 

Integration  Demonstration 

Production 

Of  O  i 

PDR  |  CDR 

i 

Knowledge  Point  1 
Technologies,  time,  funding  and 
other  resources  match  customer  needs. 

Decision  to  invest  in  product  development. 


Knowledge  Point  2 
Design  is  stable  and  performs 
as  expected. 

Decision  to  start  building  and  testing 
production  representative  prototypes. 


Knowledge  Point  3 
Production  meets  cost,  schedule, 
and  quality  targets. 

Decision  to  produce  first  units  for 
customer. 


•  Model  provides  framework  for  incremental,  time  certain  (development  constrained  to  5  to  6  years 
or  less),  and  knowledge-based  approach  to  weapon  system  acquisitions. 

•  Success  requires  structured,  disciplined  application  and  adherence  to  model. 

•  Knowledge  points  align  with  key  investment  inflection  points. 

•  Controls  are  in  place  for  decisions  makers  to  measure  progress  against  specific  criteria  and 
ensure  managers  capture  key  knowledge  before  moving  to  next  phase. 


PROGRAM  LEVEL 


Accountability  *  Integrity  *  Reliability 


Programs  Conducting  Early  Systems 
Engineering  Have  Better  Outcomes 


•  Early  systems  engineering,  ideally  before  a  program  enters 
development,  is  critical  to  ensuring  that  requirements  are 
achievable  and  designable. 


•  We  have  previously  reported  that  programs  conducting  key 

systems  engineering  events  prior  to  development  start  experienced, 
on  average,  lower  cost  growth  and  shorter  delays  in  achieving  initial 
operational  capability. 


•  Only  1  of  the  37  programs  in  our  2010  assessment  that  held  PDR 
did  so  before  development  start.  The  remaining  programs,  on 
average,  held  the  review  30  months  after  development  start. 


•  Preliminary  design  review  now  required  prior  to  development  start 
under  the  Weapon  System  Acquisition  Reform  Act  of  2009. 


PROGRAM  LEVEL 


Accountability  *  Integrity  *  Reliability 


Newer  Programs  GAO  Assessed  Are  Starting 
with  Higher  Levels  of  Technology  Maturity 


•  Since  2003,  there  has  been  an 
increase  in  the  maturity  of  critical 
technologies  at  development  start. 


•  All  6  programs  entering  system 
development  from  2006  to  2009 
had  their  critical  technologies 
demonstrated  in  at  least  a  relevant 
environment,  in  accordance  with 
the  DOD  and  statutory  criteria. 


•  However,  only  4  of  the  29 

programs  in  our  2010  assessment 
that  provided  data  started 
development  with  fully  mature 
critical  technologies.  ' 


Maturity  of  Critical  Technologies  at  Milestone  B 


Percent 


(14) 


(9) 


(6) 


Year  program  entered  system  development 


[  |  Mature 

|  |  Nearing  maturity 

Source:  GAO  analysis  of  DOD  data. 


Note:  Number  of  programs  and  technologies  in  parentheses. 


7 


PROGRAM  LEVEL 


Accountability  *  Integrity  *  Reliability 


Programs  Holding  Design  Reviews  in  Recent 
Years  Reported  Having  More  Knowledge 


•  Since  2003,  the  average 
percentage  of  design  drawings 
releasable  for  programs  at  the 
critical  design  review  has  steadily 
increased. 


•  However,  designs,  on  average,  are 
still  far  from  stable  and  concurrent 
technology  development  increases 
the  risk  subsequent  design 
changes  and  rework. 

•  Of  the  28  programs  in  our  201 0 
assessment  that  held  a  critical 
design  review,  only  8  reported 
having  a  stable  design. 


Average  Percent  of  Releasable  Design  Drawings 
at  Critical  Design  Review 

Percent 


Stable  design 

69.86% 

(19) 

51.24% 

(2) 

57.42% 

(7) 

2003  2004-  2006- 

or  prior  2005  2009 


Year  of  critical  design  review 

Source:  GAO  analysis  of  DOD  data. 


Note:  Number  of  programs  in  parentheses. 


8 


PROGRAM  LEVEL 


Accountability  *  Integrity  *  Reliability 


Programs  Are  Not  Testing  Prototypes  in  Time 
to  Prove  Out  Designs  and  Performance 


•  Programs  should  test  integrated  prototypes  before  critical  design  review  to 
demonstrate  that  the  design  is  capable  of  meeting  requirements. 


•  Only  4  of  33  programs  in  our  2010  assessment  tested  or  planned  to  test  an 
integrated  prototype  before  critical  design  review.  The  remaining  programs 
reported  that  they  will  test  these  prototypes,  on  average,  31  months  later. 


Year  of  critical  design  review 

2003  or  prior 

2004-2005 

2006-2009 

2010  or  later 

All  programs 

Number  of  programs  testing  before  critical 
design  review 

1 

0 

3 

0 

4 

Number  of  programs  testing  after  critical 
design  review 

3 

6 

15 

5 

29 

For  programs  testing  after  critical  design 
review,  average  number  of  months  from 
design  review  to  prototype  test 

74 

29 

27 

18 

31 

Source:  GAO  analysis  of  DOD  data. 
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PROGRAM  LEVEL 


Accountability  *  Integrity  *  Reliability 


Programs  Are  Not  Demonstrating  Performance  or 
Manufacturing  Processes  Before  Production 


•  Programs  should  test  production 
representative  prototypes  before 
production  start  and  bring  critical 
manufacturing  processes  into 
statistical  control. 


•  Only  17  of  the  31  programs  in  our 
2010  assessment  that  reported  a 
production  date  have  tested  or 
planned  to  test  a  production 
representative  prototype  before 
production. 

•  Only  7  programs  in  our  201 0 
assessment  had  identified  their 
critical  manufacturing  processes. 


Programs  Testing  Production  Representative 
Prototype  Before  and  After  a  Production  Decision 


prior  later 

Programs  testing  prototype  before  production  decision 
Programs  not  testing  prototype  before  production  decision 
Source:  GAO  analysis  of  DOD  data. 


PROGRAM  LEVEL 


Accountability  *  Integrity  *  Reliability 


Observations  on  Other  Factors  That  Can 
Affect  Program  Execution 


•  Requirements:  Of  the  42  programs  in  our  2010  assessment,  23  programs 
reported  at  least  one  change  to  a  key  performance  parameter  and  9 
programs  experienced  at  least  one  change  to  a  key  systems  attribute  since 
development  start. 


•  Software:  Seventeen  of  the  28  programs  that  reported  software  data 
estimated  that  software  lines  of  code  had  grown  by  25  percent  or  more 
since  development  start.  The  average  lines  of  code  growth  was  92  percent. 


•  Program  office  staffing:  Nineteen  of  50  programs  that  responded  to  our 
staffing  questions  were  able  to  fill  all  authorized  positions.  As  a  result, 
program  offices  reported  that  program  management  and  oversight  had 
been  degraded,  contracting  activities  had  been  delayed,  and  program 
management  costs  had  increased  as  contractors  were  used  to  fill  gaps. 


PROGRAM  LEVEL 


Accountability  *  Integrity  *  Reliability 


Observations  on  Other  Factors  That  Can 
Affect  Program  Execution 


•  Programs’  reliance  on  non-governmental  personnel  continues  to  increase 
in  order  to  make  up  for  shortfalls  in  government  personnel  and  capabilities. 


Program  Office  Composition  for  50  DOD  Programs 


Program 

management 

Engineering 
and  technical 

Contracting 

Other 

business 

functions 

Administrative 

support 

Other 

Total 

Military 

28% 

7% 

6% 

3% 

2% 

5% 

8% 

Civilian 

government 

40% 

41% 

74% 

45% 

18% 

24% 

40% 

Total 

government 

67% 

47% 

80% 

48% 

20% 

29% 

49% 

Support 

contractors 

32% 

43% 

20% 

50% 

78% 

70% 

45% 

Other 

nongovernment 

0% 

9% 

0% 

3% 

2% 

1% 

6% 

Total 

nongovernment 

33% 

53% 

20% 

52% 

80% 

71% 

51% 

Note:  Totals  may  not  add  due  to  rounding. 
Source:  GAO  analysis  of  DOD  data. 
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ACQUISITION  REFORMS 


Accountability *  *  Integrity  *  Reliability 


New  DOD  Policies  Could  Improve  Outcomes 


•  Recent  department  initiatives  may  help  focus  on  joint  warfighter  needs 

•  Functional  COCOMs  given  greater  voice  in  requirements  process. 

•  Preferred  materiel  solutions  no  longer  identified  in  initial  capability  proposals, 
giving  greater  emphasis  to  trade-off  analysis  via  AOAs. 

•  New  Capability  Portfolio  Management  framework  could  facilitate  more 
strategic  investment  choices 

•  Portfolio  managers  provided  key  input  in  recent  budget  cycle,  but  they  do  not 
have  decision-making  authority. 

•  More  discipline  and  up-front  knowledge  in  early  acquisition  phases  could  put 
programs  on  more  stable  footing 

•  Early  Materiel  Development  Decision  required  for  all  programs. 

•  Preference  for  incremental  development,  with  baselines  for  each  increment. 

•  PDR  required  before  system  development  start. 

•  Competitive  prototyping  required  as  part  of  technology  development  phase. 

•  Configuration  Steering  Boards  established  to  control  requirements  creep. 


ACQUISITION  REFORMS 


Accountability *  *  Integrity  *  Reliability 


Weapon  System  Acquisition  Reform  Act 
Stresses  Cost  and  Performance  Assessment 


•  Elevates  the  role  of  cost  assessment  and  program  evaluation 

•  Duties  of  Director,  Cost  Assessment  and  Program  Evaluation  (CA&PE)  include 
reviewing  cost  estimates,  conducting  independent  cost  estimates,  and 
approving  the  choice  of  baseline  cost  estimates  for  all  ACAT  ID  programs. 

•  Director,  CAPE,  will  also  formulate  study  guidance  for  analysis  of  alternatives. 


•  Emphasizes  importance  of  systems  engineering  and  developmental  test 

•  Established  Directors  of  Developmental  Test  and  Evaluation  and  Systems 
Engineering  to  oversee  policy  and  guidance  and  approve  test  and  systems 
engineering  plans. 

•  Requires  services  to  periodically  assess  their  capabilities  in  these  areas. 


•  Increases  oversight  and  reporting  on  cost  estimating,  systems  engineering, 
developmental  test,  program  performance,  and  technology  maturity. 


•  Stresses  importance  of  competition  throughout  the  acquisition  cycle. 


ACQUISITION  REFORMS 


Accountability  *  Integrity  *  Reliability 


Programs  Have  Begun  to  Implement  DOD’s 
Revised  Acquisition  Policies 


•  Programs  in  our  2010  assessment  have  begun  to  implement 

acquisition  reforms  that  could  improve  cost  and  schedule  outcomes. 

•  Competitive  prototyping  -  8  of  1 0  pre-major  defense  acquisition 
programs  in  our  assessment  reported  planning  to  develop  competitive 
prototypes  of  the  proposed  weapon  system  or  key  system  elements 
prior  to  Milestone  B. 

•  Early  systems  engineering  -  7  of  1 0  pre-major  defense  acquisition 
programs  in  our  assessment  have  already  scheduled  a  preliminary 
design  review  before  Milestone  B. 


•  Only  a  few  programs  in  our  2010  assessment  reported  holding 
configuration  steering  board  meetings  in  2009. 

•  For  7  programs  that  held  meetings  in  2009,  none  reported  that  the 
board  approved  a  requirement  change  or  significant  technical  change. 

•  One  program  presented  de-scoping  options  to  the  board  and  had 
those  approved  to  help  maintain  cost  and  schedule. 


Backup  Slides 
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GAO 

Accountability  *  Integrity  *  Reliability 


CONDITION 


Accountability  *  Integrity  *  Reliability 


DOD  Investment  Levels  Remain  High, 
But  Are  Not  Sustainable 


FY  81  FY  84  FY  87  FY90  FY  93  FY  96  FY  99  FY  02  FY  05  FY  08  FY11  FY 14 

Research,  Development,  Test  and  Evaluation  and  Procurement  Funding 


Source:  GAO  analysis  of  National  Defense  Budget  Estimates  for  the  Fiscal  Year  201 1  Budget. 
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CONDITION 


Accountability  *  Integrity  *  Reliability 


DOD’s  Major  Defense  Acquisition  Portfolio 
Has  Experienced  Poor  Outcomes 


Performance  of  DOD’s  Major  Defense  Acquisition  Program  Portfolio 


Portfolio  status 

Fiscal  year  2003 

Fiscal  year  2007 

Fiscal  year  2008 

Number  of  programs 

77 

95 

96 

Total  planned  commitments 

$1 .2  trillion 

$1 .6  trillion 

$1 .6  trillion 

Commitments  outstanding 

$724  billion 

$875  billion 

$786  billion 

Change  to  total  RDT&E  costs  from  first  estimate 

37  percent 

40  percent 

42  percent 

Change  in  total  acquisition  cost  from  first  estimate 

19  percent 

26  percent 

25  percent 

Estimated  total  acquisition  cost  growth 

$183  billion 

$301  billion 

$296  billion 

Share  of  programs  with  25  percent  or  more  increase 
in  program  acquisition  unit  cost 

41  percent 

44  percent 

42  percent 

Average  delay  in  delivering  initial  capabilities 

18  months 

21  months 

22  months 

Note:  Analysis  was  not  updated  for  the  fiscal  year  2009  because  DOD  did  not  issue  Selected  Acquisition  Reports. 


Source:  GAO  analysis  of  December  2007  Selected  Acquisition  Reports. 
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CONDITION 


Accountability  *  Integrity  *  Reliability 


New  Programs  Are  Performing  Better  Than 
Older  Programs 


Changes  in  Program  Cost  and  Schedule  by  Age  of  Program 
Fiscal  Year  2008  Portfolio 


Age  of  Program 

Change  in  total 
RDT&E  costs 
from  first 
estimate 

Change  in  total 
acquisition  cost 
from  first 
estimate 

Average 
change  in 
quantities 

Average 
number  of 
months  late 

Number  of 
programs 

15  or  more  years  since 
development  start 

47  percent 

19  percent 

-39  percent 

37  months 

10 

10  to  14  years  since 
development  start 

73  percent 

53  percent 

52  percent 

26  months 

17 

5  to  9  years  since 
development  start 

37  percent 

31  percent 

9  percent 

22  months 

25 

Less  than  5  years 
since  development  start 

12  percent 

11  percent 

1  percent 

5  months 

28 

Note:  Analysis  was  not  updated  for  the  fiscal  year  2009  because  DOD  did  not  issue  Selected  Acquisition  Reports. 


Source:  GAO  analysis  of  December  2007  Selected  Acquisition  Reports. 
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CONDITION 


Accountability  *  Integrity  *  Reliability 


Promised  Capabilities  Are  Delivered  Later 
Than  Planned 


Schedule  Delays  for  DOD’s  2008  Program  Portfolio 


Programs  planning  to  achieve  IOC 
on  time  (or  less  than  1  month  late) 
(20  programs) 


Programs  planning  to  achieve  IOC 
between  1  to  12  months  late 
(17  programs) 


Programs  planning  to  achieve  IOC 
between  13  to  24  months  late 
(13  programs) 


Programs  planning  to  achieve  IOC 
between  25  to  48  months  late 
(12  programs) 

Programs  planning  to  achieve  IOC 
more  than  48  months  late 
(10  programs) 


Note:  Analysis  was  not  updated  for  the  fiscal  year  2009  because  DOD  did  not  issue  Selected  Acquisition  Reports. 
Source:  GAO  analysis  of  December  2007  Selected  Acquisition  Reports. 
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CONDITION 

Consequences  of  Poor  Outcomes 


k  GAO 

Accountability  *  Integrity  *  Reliability 


Cost  Growth 


I  l 

][ 

t 


DOD  must  request  more  funding  to 
cover  cost  overruns,  make  trade¬ 
offs  with  existing  programs,  delay 
the  start  of  new  programs,  or  take 
funds  from  other  accounts 


Means  less  funding 
for  other  priorities 


Reduces  DOD’s 
buying  power 


Schedule  Delays 


l 


Critical  capabilities  not  provided 
to  warfighter  when  needed 


DOD  must  operate  costly  legacy 
systems  longer  than  expected,  find 
alternatives  to  fill  capability  gaps,  or 
go  without  a  capability 
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GAO 

CONTRIBUTING  FACTORS  Accountability  *  Integrity  *  Reliability 

Program  Managers  Cited  Requirements  and 
Funding  as  Biggest  Obstacles  to  Success 


Funding  Instability  Requirements  Staffing  Oversight 

Instability 


Source:  GAO-06-1 12SP. 
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ap.  Gr  A  O 

CONTRIBUTING  FACTORS  Accountability  *  Integrity  *  Reliability 

Strategic  and  Program  Level  Factors  Lead  to 
Poor  Outcomes 


STRATEGIC  LEVEL  PROGRAM  LEVEL 


•  DOD’s  processes  for 
determining  warfighter  needs, 
allocating  resources,  and 
managing  acquisition  programs 
are  fragmented 

•  Success  defined  in  terms  of 
starting  and  sustaining 
programs 


•  Programs  are  proposed  and 
approved  without  adequate 
knowledge  about 
requirements  and  resources 
needed  to  execute  them 

•  Managers  rely  on  optimistic 
assumptions  about 
requirements,  technologies, 
cost,  and  schedule 
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PROGRAM  LEVEL 


Accountability  *  Integrity  *  Reliability 


Programs  that  Changed  Key  Requirements 
Experienced  Added  Instability 


•  For  programs  in  our  2009  assessment  that  had  at  least  one  requirements 
change,  the  average  RDT&E  cost  was  more  than  three  times  higher  and 
the  average  schedule  delay  was  twice  as  long  as  programs  without  these 
changes." 


Percentage  of  change  in  RDT&E  cost 


requirements  requirements 

changes  changes 


Months 


requirements  requirements 

changes  changes 


Source:  GAO  analysis  of  DOD  data. 


24 


DOD  ACQUISITION  POLICY 


Accountability  *  Integrity  *  Reliability 


Are  Effective  Controls  In  Place  to  Ensure  a 
Knowledge-Based  Approach  is  Followed? 


Knowledge  Point  1 :  Technologies,  time,  funding,  and  other  resources  match  customer  needs.  Decision  to 
invest  in  product  development. 

Types  of  controls  considered  effective  for  ensuring  knowledge-based  approach 

Are  controls  present  in  DOD  5000  policy? 

Demonstrate  technologies  to  a  high  readiness  level  -  technology  readiness  level  7  -  to  ensure 
technologies  will  work  in  an  operational  environment 

1 

Ensure  that  requirements  for  product  increment  are  informed  by  preliminary  design  review  using 
systems  engineering  process  (such  as  prototyping  of  preliminary  design) 

X 

Establish  cost  and  schedule  estimates  for  product  on  the  basis  of  knowledge  from  preliminary 
design  using  system  engineering  tools  (such  as  prototyping  of  preliminary  design) 

X 

Constrain  development  phase  (5  to  6  years  or  less)  for  incremental  development 

1 

Ensure  development  phase  fully  funded  (programmed  in  anticipation  of  milestone) 

1 

Align  program  manager  tenure  to  complete  development  phase 

1 

Conduct  independent  cost  estimate 

X 

Conduct  independent  program  assessment 

Conduct  major  milestone  decision  review  for  development  start 

X 

X  =  Sufficient  controls  in  place.  Controls  are  considered  effective  if  they  are  backed  by  specific  criteria  required  for  review,  and  if  decision 
makers  are  required  to  consider  the  resulting  data  before  deciding  to  advance  a  program  to  the  next  level. 

I  =  Intent  identified  in  the  policy,  but  sufficient  controls  not  in  place  by  knowledge  point. 


DOD  ACQUISITION  POLICY 


Accountability  *  Integrity  *  Reliability 


Are  Effective  Controls  In  Place  to  Ensure  a 
Knowledge-Based  Approach  is  Followed? 


Knowledge  Point  2:  Design  is  stable  and  performs  as  expected.  Decision  to  start  building  and  testing 
production-representative  prototypes. 

Types  of  controls  considered  effective  for  ensuring  knowledge-based  approach 

Are  controls  present  in  DOD  5000  policy? 

Complete  system  critical  design  review 

X 

Complete  90  percent  of  engineering  design  drawing  packages 

Complete  subsystem  design  reviews 

1 

Demonstrate  with  system  level  integrated  prototype  that  design  meets  requirements 

Complete  the  failure  modes  and  effects  analysis 

1 

Identify  key  system  characteristics 

1 

Identify  critical  manufacturing  processes 

1 

Establish  reliability  targets  and  growth  plan  on  the  basis  of  demonstrated  reliability  rates  of 
components  and  subsystems 

1 

Conduct  independent  cost  estimate 

Conduct  independent  program  assessment 

Conduct  major  milestone  decision  review  to  enter  system  demonstration 

1 

X  =  Sufficient  controls  in  place.  Controls  are  considered  effective  if  they  are  backed  by  specific  criteria  required  for  review,  and  if  decision 
makers  are  required  to  consider  the  resulting  data  before  deciding  to  advance  a  program  to  the  next  level. 

I  =  Intent  identified  in  the  policy,  but  sufficient  controls  not  in  place  by  knowledge  point. 


DOD  ACQUISITION  POLICY 


Accountability  *  Integrity  *  Reliability 


Are  Effective  Controls  In  Place  to  Ensure  a 
Knowledge-Based  Approach  is  Followed? 


Knowledge  Point  3:  Production  meets  cost,  schedule  and  quality  targets.  Decision  to  produce  first  units  for 
customer. 

Types  of  controls  considered  effective  for  ensuring  knowledge-based  approach 

Are  controls  present  in  DOD  5000  policy? 

Demonstrate  manufacturing  processes 

1 

Build  and  test  production-representative  prototypes  to  demonstrate  product  in  intended 
environment 

1 

Test  production-representative  prototypes  to  achieve  reliability  goal 

Collect  statistical  process  control  data 

Demonstrate  that  critical  processes  are  capable  and  in  statistical  control 

Independent  cost  estimate 

X 

Independent  program  assessment 

Conduct  major  milestone  decision  review  to  begin  production 

X 

X  =  Sufficient  controls  in  place.  Controls  are  considered  effective  if  they  are  backed  by  specific  criteria  required  for  review,  and  if  decision 
makers  are  required  to  consider  the  resulting  data  before  deciding  to  advance  a  program  to  the  next  level. 

I  =  Intent  identified  in  the  policy,  but  sufficient  controls  not  in  place  by  knowledge  point. 


